DOCUMENT RESUME 



ED 452 161 



SP 039 892 



AUTHOR 

TITLE 

INSTITUTION 
PUB DATE 
NOTE 



PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



Rudy, Dennis W. 

What Matters Most: HealthWorks ! Kids’ Museum Annual 
Evaluation Report of Findings, Year 1 of 5 . 

HealthWorks! Kids’ Museum, South Bend, IN. 

2001-01-31 

3 5p . ; HealthWorks! Kids' Museum is sponsored by Memorial 
Health Sysems and the Memorial Health Foundation, along with 
funding by the Harry and Jeannette Weinberg Foundation, the 
Memorial Hospital Auxiliary, and the Junior League of South 
Bend. An executive summary is appended. 

Reports - Evaluative (142) -- Tests/Questionnaires (160) 

MF01/PC02 Plus Postage. 

*Child Health; Community Programs; Creative Teaching; 
Elementary Secondary Education; *Health Education; *Health 
Promotion; Museums; Outreach Programs; Program 
Effectiveness; Program Evaluation 



ABSTRACT 



This report presents an evaluation of the HealthWorks! Kids' 
Museum, an urban education center designed to help children in grades preK-8 
understand and make good choices about healthy living and lifestyle choices. 
It includes an exhibit floor and interactive classroom areas with a program 
highlighting how body systems work; a game challenging the senses and 
intelligence; a computer kiosk with health-related questions; a climbing wall 
that looks like magnified human skin; and video games. The evaluation plan 
uses multiple data measures, encompassing both content and process concerns, 
formative and summative assessment, and on-site and off -site activities. 
Qualitative and quantitative measures are utilized. Data collection includes 
surveys of the visiting public, teacher evaluations, and artifact reviews. 
Written documentation regarding all assessment activities are given in an 
annual report. A summative assessment of all data related to the program is 
compiled in a longitudinal report. Evaluation documentation are presented to 
stakeholders at annual meetings . Findings are disseminated to the larger 
professional health community. Year 1 findings indicate that children enjoy 
and use the interactive activities and consider them meaningful and fun. 
Teachers, parents, and community members report increased awareness and 
knowledge of health issues. An executive summary is appended. (Contains 10 
references.) (SM) 
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Introduction 

Program Overview 

HealthWorks! Kids' Museum, which opened on 
February 12, 2000, is a 12,000 square foot 
innovative education center located within the 
Memorial Leighton HealthPlex in downtown 
South Bend, IN. It is designed to help children in 
grades PreK-8 understand and make good choices 
about healthy living and lifestyle choices. The 
museum is open daily for school groups, and 
evenings and weekends for community groups 
and families. 

The museum was proposed, designed and 
implemented in response to the needs of the 
community. Countless community members, 
volunteers and staff professionals worked with 
consultants and specialists from across the nation 
in determining program offerings, exhibit designs, 
and instructional methods appropriate for the 
museum. 

The museum consists of an exhibit floor and interactive classroom areas. Some 
features of the HealthWorks! Kids' Museum currently include: 

• BodyWorks! - highlighting body systems and how they work, 

• The Main Brain theatre - an exploration into the mind, 

• MindWorks! - challenging the senses and intelligence, 

• All About Me - kiosks of computers focusing on health-related 
questions, 

• The Skin/ Crawl Wall - a professional climbing wall that looks like 
magnified human skin, 

• Virus Invaders - a live action video game where one fights off bacteria 
and viruses, 

• Interactive Video Classrooms - for demonstrations and group activities, 
and 

• Resource Center - offering reference materials for students, teachers and 
families. 

The HealthWorks! Kids’ Museum is just one of many community health outreach 
efforts sponsored by Memorial Health Systems and the Memorial Health 
Foundation, such as bicycle safety, anti-smoking, and sexual abstinence 
programs for students in elementary, middle and high schools. The doctors and 
staff of Memorial Hospital of South Bend have a long and rich history of 
community involvement within the Michiana region. 



Organizations like ours 
try to learn from our 
experiences, both the 
successful and the not so 
succesful ones. This is a 
way of assessing our 
effectiveness and 
sharing information. 



Phil Newbold 
CEO, Memorial 
Hospital / Health System 
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The project is sponsored by Memorial Health Systems and the Memorial Health 
Foundation, along with funding by the Harry and Jeannette Weinberg 
Foundation, Dennis and Mary Lou Schwartz, James and Julia Schwartz, the 
Shields and Warner families, the Memorial Hospital Auxiliary, and the Junior 
League of South Bend. 



Spheres of Influence 

The founders of HealthWorks! envision an ever widening sphere of influence 
beginning with the museum activities. It is hoped that once a child attends the 
museum, she or he will want to share the museum experience with others 
important in their lives. Thus, it is believed that family members and others at 
school (i.e. teachers, staff & students) will be made aware of healthy living ideas 
and opportunities for them in which to share. 

Eventually this ever widening sphere of influence would impact the community 
in positive ways related to healthy living. The diagram below describes this 
three-tier model of influence, with Sphere 1 related to children who attend 
HealthWorks! Kids' Museum, Sphere 2 consisting of family and school members, 
and Sphere 3 representing the larger community of Michiana. 




Three 
Sphere 
Model of 
Influence 



Program Goals 

Through much deliberate planning and effort by various oversight committees, 
program goals were posed, revised and finalized to guide the curricular and 
instructional efforts of the museum. For example, the following four program 
goals finalized by the Reach & Teach Advisory Group (RTAG) committee were 
formally adopted on July 28, 1998 to be part of the evaluation plan. The 
overriding program goals are: 
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Children will : 

• engage in interactive processes of discovery that will stimulate the 

curiosity, and result in new exploration, knowledge, and positive 
attitudes about healthy living, 

• be intrigued and empowered to explore the many choices and 

options they have and will develop the resources and skills to make 
informed choices in their daily living, 

• appreciate the wonder and complexity of the human body, its senses, 

intelligence and spirit, and 

• remember the visit as fun, exciting and meaningful, will extend their 

exploration of healthy living beyond their experience, and will 
return with family and friends. 

It was the decision of the oversight committees to recommend further program 
objectives to be written by the HealthWorks! staff as necessary to ensure 
implementation of the program goals as curricula and program offerings are 
developed. Currently three distinct program offerings are available for selection 
by school and community groups visiting HealthWorks! Kids' Museum. Each 
has distinct written objectives tied to the larger program goals. 

Current program offerings in Year 1 of the museum operation are: 

Safety: You've Got What It Takes! 

Students review common personal safety rules through role-playing, 
demonstration and games. 

Mission Possible: A Healthy Body ! 

Students investigate nutrition, exercise, rest, and attitude as they strive to 
uncover the mystery of well-being. 

The Amazing Body 

Students examine the heart, lungs, brain, bones and muscles to determine 
which is the body superstar through working on teams. 

Evaluation Purpose 

The evaluation purports to assess the extent to which each of the program goals 
has been met and to also indicate the corresponding merit and worth of the 
stated goals. Sometimes the evaluation activities are formative in nature, taking 
place during the actual implementation phase of the project and providing 
feedback for program improvements. Other evaluation techniques are designed 
strictly to be summative, occurring at checkpoints along the duration of the 
project and at the end of the project, thus providing commentary about merit and 
worth of the program (Worthen & Sanders, 1987). 

Ensuring that evaluation activities address both formative and summative 
aspects of the program being studied is an actual strength of the design of this 
project and any evaluation study. 
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Evaluation Plan & Procedures 



Design of the Study 

A good program evaluation tells a story. If it's 
told well, it's a story of personal striving, collected 
efforts and good intentions. Through the story we 
might come to know the shared vision of an 
organization, what life is like within a certain 
community or discover new ideas that can 
positively impact the lives of others. 

The evaluation and documentation of human 
activity can be complex, time-consuming, and 
somewhat messy. For what can be more complex 
than attempting to document the depth and 
breadth of intended and unplanned human 
learning? That is what makes evaluation so 
intriguing. 




Engaged on the Skin 
Crawl Wall at 
HealthWorks! 



A well constructed evaluation plan can document or confirm our research 
questions, and yet reveal new learnings and new questions that we never even 
considered. When evaluation studies are conducted in the true spirit of inquiry, 
insights and new learnings will inevitably occur. 



O 
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Mixed-methods Evaluations 

Latest findings from the research community support a mix of qualitative and 
quantitative data sources when conducting program evaluation research 
(Frechtling & Sharp, 1997). A mix of both qualitative and quantitative data is 
being collected to assess the knowledge, behaviors and attitudes of various 
visitors and groups of individuals that frequent the HealthWorks! Kids’ 
Museum. 

This blend of evaluative methodologies yields a rich database that can be used 
for short-term feedback and formative assessment of various phases of the 
project, as well as long-term planning and summative assessment of the merit 
and worth of the museum's activities upon the community (Denzin & Lincoln, 
1994). 

Confirming Evidence Approach 

The evaluation plan utilizes a confirming evidence approach to gauge program 
effectiveness, and provide insight and understanding for continued planning of 
the partnership activities for the duration of the project. For example, each 
program goal and research question addressed in the evaluation plan is studied 
both formatively and summatively, thus providing feedback on current program 
efforts and also providing recommendations for addressing long-term planning 
of specific issues and concerns (Patton, 1990). 
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Each program goal is said to have been achieved or met by Confirming Evidence, if 
and when multiple data measures confirmed same. Program goals and research 
questions which produce conflicting or differing pieces of data measures are said 
to produce Mixed Evidence, and are deemed inconclusive. Disconfirming Evidence 
is the term used when multiple data measures verifiy that program goals have 
not been successfully met (Rudy, 1999a; 1999b). 

Triangulation 

A method of data collection used by many researchers to ensure accurate 
findings is called triangulation. Basically, triangulation requires that multiple 
data measures (i.e. minimally three distinct measures) be used to produce 
evidence related to each of the questions studied or addressed in the evaluation 
plan (Miles & Huberman, 1984). 



For example, if one was interested in verifying 
whether or not a visit to a museum had significant 
positive impact on the attitude of a student 
towards healthy living, more than one piece of 
evidence would be required to confirm or 
disconfirm this belief. In this situation one might 
ask the student to complete a survey regarding 
healthy living, but also observe how the student 
chooses to spend his/her time while at the 
museum along with possibly asking a classroom 
teacher to corroborate the students' choices in a 
school setting. This variety and multiplicity of 
data measures strengthens the validity and 
reliability of any conjectures ultimately made 
about the student's attitude toward healthy living. 




Logging some miles on a 
virtual tour in the country 
via bicycles. 



The evaluation plan utilizes triangulation along 
with the confirming evidence approach as the 
basis of all evaluation activities. 



ERIC 



Data Collection Plan 

The evaluation plan encompasses both content and process concerns (i.e. health 
content knowledge and behavior; attitude, instruction), formative and 
summative assessment (i.e. short-term and long-term planning /feedback), and 
on-site and off-site activities (i.e. health center and non-museum activities). A 
myriad of data measures are to be utilized, some naturally occurring in the 
project environment and some instrumentation that will have to be created and 
implemented. Remember that triangulation is to be utilized to verify the extent 
to which program goals are met, so data will be abundant and complex. 

Several broad categories of data measures were utilized in data collection and 
data analyses. Again both qualitative and quantitative measures were utilized to 
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assess the extent to which program efforts successfully addressed the three foci 
of students, staff and parents described in the major project goal. 

The following performance indicators were selected and deemed appropriate for 
use in the study by the project staff and oversight committees. The indicators are 
categorized by the three spheres of influence: 

Sphere 1 Children 

Observations of Students at HealthWorks! 

Student Surveys 
Student Artifacts 
Student Feedback 

Student Performance at HealthWorks! and in Related Classroom 
Activities 

Sphere 2 Family & Teachers 

Survey of Saturday Visitors 

Feedback from Parents/ Families Whose Children Attended 
HealthWorks! 

Teacher Surveys 
Teacher Artifacts 
' Teacher Feedback 

Sphere 3 Community 
Community Surveys 
Community Feedback 
Community Artifacts 

Community Health Related Data (trends, patterns, pre-post) 

Cohort Group Design 

In addition to the data sets described above, a cohort group of frequent visitors 
to the museum has been planned. This group, composed of students from 
grades 3 -8 from a neighboring public school district, has committed to visit the 
museum annually. Data from Students attending the museum will be collected 
based upon student knowledge, attitude and behavior. This data will be 
collected longitudinally for five years. 

In year five of the study this cohort group data will be utilized as a comparison 
or experimental group for study compared to like groups from within the 
Michiana region. This data will be analyzed in a pretest-posttest gainscore 
design, along with other qualitative and quantitative measures to assess the 
extent to which repeated trips to the museum impact the healthy lifestyles of the 
community. 




Instrumentation & Analyses 

Several broad categories of data measures will be utilized in data collection and 
data analyses. Some of these data measures require the development and 
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implementation of unique contextually based data collection measures. Some of 
the data measures include, but are not limited to the following: 

1) Teacher/ Evaluator OBSERVATION 

(e.g. protocol rating sheet; checklist; videotaped lesson) 

One example of this data measure is actual observation of students by 
teacher/ evaluator during the activity related to previously established 
criteria. 

2) Teacher/ Evaluator REFLECTION 

(e.g. teacher judgement; anecdotal records; journal) Measures of teacher 
reflection may include judgements and comments made about a student 7 s 
performance and/or level of understanding after an activity/ episode has 
been completed based upon previously stated criteria. 

3) Teacher /Evaluator ARTIFACT 

(e.g. course syllabus; problem-based learning activity; home web page; 
attendance record) Some artifacts of choice are items of record produced 
by the instructor utilized during the teaching and/or learning 
activity/ episode based upon predetermined criteria for student 
performance. 

4) Student/ Participant/ Museum Visitor REFLECTION 

(e.g. student journal; structured interview; survey) Some reflective 
measures to be considered for use in the study include self-assessment 
and appraisal of performance and/or level of understanding (prior 
student training, practice and discussion of assessment criteria is 
necessary to use this procedure). 

5) Student / Participant / Museum V isitor 
ARTIFACT 



(e.g. contents of a portfolio; lesson plan; 
essay; criterion-referenced test; group 
project; concept map). Some examples of 
student data utilized as artifacts are 
assessments completed based upon review 
and analysis of student's actual work; items 
produced through participation in specific 
learning activities. Assessment criteria 
must be established prior to evaluation of 
materials; portfolios must be available for 
future reference and data verification (i.e. 
may be stored on CD-ROM, computer 
diskette, etc.); concept map completed at 
the health center or off-site (Ruiz-Primo, 
Schultz & Shavelson, 1990; Trochim, 1989). 




Exploring the now 
famous pig lungs. 
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6) Student/ Participant/ Museum Visitor PEER REVIEW 

(e.g. observational checklist; performance rating sheet; feedback from an 
observer) These data measures include peer-assessment/ appraisal of 
another student's performance and/or level of understanding (prior 
student training, practice and discussion of assessment criteria is 
necessary to use this procedure). 

The above-mentioned data points will be used to measure various cognitive, 
attitudinal and behavioral facets of the children and community members who 
visit the HealthWorks! Kids’ Museum. 

Again, triangulation of all program goals and research questions will yield 
multiple data measures. The confirming evidence approach produces 
documentation verifying one of the following findings for each item of analyses: 
Confirming Evidence; Mixed Evidence; or, Disconfirming Evidence. This 
approach is being utilized with qualitative data for it is appropriate and provides 
the information needed to inform the various stakeholders (Joint Committee on 
Standards for Educational Evaluation, 1981). 

When quantitative data is to be analyzed, an appropriate and suitable statistic of 
choice will be utilized. The actual methodology employed will be determined by 
the Evaluation Team, once program goals are finalized and instrumentation 
concerns are addressed. 



O 
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Validity & Reliability Issues 
Appropriateness (Validity) 

The appropriateness of the selected qualitative and quantitative assessments and 
their related administration procedures were reviewed for validity. The 
qualitative measures were determined to be valid if the utility principle was 
evidenced (Joint Committee on Standards for Educational Evaluation, 1981). 
Validity for the quantitative measures was evidenced through face/content 
documentation provided by the publisher or determined by staff agreement. 

In general, the qualitative measures used to gauge project activities are: 

• teacher artifacts (e.g. curriculum planning documents; instructional 
units), 

• student artifacts (e.g. samples of student work), 

• teacher feedback (e.g. focus group feedback), 

• student feedback (e.g. focus group feedback), 

• parent feedback (e.g. focus group feedback), and 

• community feedback (e.g. focus groups). 

The quantitative measures selected and used within the evaluation study 
included: 

• student performance on related knowledge activities, 

• student attitude surveys, 
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• teacher surveys, 

• parent surveys. 

• community surveys, and 

• community indicators of healthy living. 



Consistency (Reliability) 

Consistency of assessment results over time was used as the operational 
definition of reliability. For this study, quantitative measures would be deemed 
reliable if student performance was consistently measured over time. Since this 
is only year two of a five year study, more quantitative assessment is needed to 
determine reliability for locally developed criterion-referenced measures (CRT's) 
and performance assessments (PA's). The standardized norm-referenced 
assessments (NRT’s) have already proven to be both reliable and valid by the 
publishers. 

Regarding qualitative measures, accuracy of results constitutes reliability as 
defined by the AEA Evaluation Standards (Joint Committee on Standards for 
Educational Evaluation, 1981). For this study, all qualitative measures were 
deemed to meet this reliability criterion through staff review and consideration 
of all assessments utilized in the grant. 



Evaluation Activities 



The evaluation portion of the grant assesses the 
extent to which each of the program goals has 
been met and also indicates corresponding merit 
and worth of the stated goals. Sometimes the 
evaluation activities were strictly formative in 
nature, taking place during an implementation 
phase of the grant project and providing feedback 
for program improvements. Other evaluation 
techniques were designed to be summative in 
nature, occurring at the end of selected period of 
time and providing commentary about merit and 
worth of the program (Worthen & Sanders, 1987). 
Ensuring that evaluation activities address both 
formative and summative aspects of the program 
being studied is an actual strength of the design of 
the project and the evaluation study. 




Sorting bones at the 
kids' museum. 



Again, triangulation of all program goals and research questions resulted in 
multiple data measures. The confirming evidence approach produced 
documentation verifying one of the following findings for each item of analyses: 
Confirming Evidence; Mixed Evidence; or, Disconfirming Evidence. 
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Evaluation Timeline & Reporting 

The evaluation plan will be written in a formative and summative fashion. 
Formative evaluation activities will last from 6 months through 1 year, while 
summative evaluation activities will provide benchmarks at greater intervals of 
time (2-5 years). The project will be broken into phases to better enable that 
evaluative data be viewed both formally for short-term planning and 
summatively for long-term evaluation of program worth and merit. 

A timeline of evaluation activities follows: 





Evalauation Timeline 


Phase 1 


Formative Evaluation 

Year 1 of the Study 

January, 2000 - December 31, 2000 


Phase 2 


Summative Evaluation 
Years 1 & 2 of the Study 
January, 2000 - August, 2001 


Phase 3 


Formative Evaluation 
Year 3 of the Study 
August, 2001 - August, 2002 


Phase 4 


Summative Evaluation 
Years 3 & 4 of the Study 
August, 2001 - August, 2003 


Phase 5 


Summative Evaluation 
Years 1 - 5 of the Study 
January, 2000 - August, 2004 



Annual Reports 

Written documentation regarding all assessment activities will be documented in 
an annual report. This document will contain all formative and summative 
assessment data related to the program goals and project activities. Data will be 
displayed in regards to the lmowledge, attitudes and behaviors of museum 
visitors consistent with the philosophy and intent of HealthWorks! Kids' 
Museum. For example, the first annual report will focus on the themes of The 
Human Body, Personal Health and Injury Prevention /Safety that reflect the 
opening day curricula of the HealthWorks! Kids' Museum. 
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The Evaluation team will be responsible for submitting the annual report each 
July 1st to the project director of the HealthWorks! Kids' Museum. These reports 
will be completed and filed annually for the first five years of the project (see 
chart on Page 14 for additional information). 

Longitudinal Reports 

In addition to the yearly documentation of all assessment activities found in the 
annual reports, a summative assessment of all data related to the program goals 
and project activities will be compiled in a longitudinal report. Data will be 
displayed in regards to the knowledge, attitudes and behaviors of museum 
visitors consistent with the philosophy and intent of the HealthWorks! Kids' 
Museum. 

The Evaluation team will be responsible for 
submitting the longitudinal report on July 31, 2001 
at the end of the first two years of the project, and 
again on July 31, 2004 at the end of the project's 
fifth year. 

Report to Stakeholders 

Besides the written documentation provided 
through annual and longitudinal reports, the 
evaluation team will formally present an overview 
of assessment documentation and findings to all 
stakeholders at a publicized annual meeting. At 
this time members of Memorial Health 
Foundation, Memorial Health Systems and 
various community groups can dialogue with the 
evaluation team members regarding issues and 
concerns. Minutes of the stakeholders meeting 
will be provided for future use and consideration. 

Dissemination Plan 

Part of the responsibility of the evaluation team is to disseminate the findings of 
the study to representatives of the larger professional health community. To 
fulfill this obligation, the evaluation team will present at regional and annual 
conferences sponsored by professional organizations representing both the 
health and education communities. The evaluation team will focus on 
methodological concerns related to evaluation and also to relevant findings 
regarding health issues. 

These presentations should prove to solidify the role and vision of Memorial 
Health Systems and the Memorial Health Foundation as leaders within the field 
of community health education. 




Checking out the 
senses attached to the 
main brain. 
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Data Collection Schedule 



Clients 


Annual 

Report 

(formative) 


Biennial 

Report 

(summative) 


Annual 

Report 

(formative) 


Biennial 

Report 

(summative) 


Final 

Report 

(summative) 


February, 2000 - 
December, 2000 


February, 2000- 
August, 2001 


August, 2001- 
August, 2002 


August, 2001- 
August, 2003 


January, 2000 - 
August, 2004 


Student & 
Teachers Visiting 
HealthWorks! 
(Sphere 1) 


Observations, 

Surveys, 

Artifacts, 

Feedback 


Observations, 

Surveys, 

Artifacts, 

Feedback 


Observations, 

Surveys, 

Artifacts, 

Feedback 


Observations, 

Surveys, 

Artifacts, 

Feedback 


Observations, 

Surveys, 

Artifacts, 

Feedback 


Cohort Group of 
Repeated Visitors 
to HealthWorks! 
(Sphere 1) 


Observations, 

Surveys, 

Artifacts, 

Feedback, 

Performance 


Observations, 

Surveys, 

Artifacts, 

Feedback, 

Performance 


Observations, 

Surveys, 

Artifacts, 

Feedback, 

Performance 


Observations, 

Surveys, 

Artifacts, 

Feedback, 

Performance 


Observations, 

Surveys, 

Artifacts, 

Feedback, 

Performance 


Extended Family 
Members 
(Sphere 2) 


< -. f y.- -• ^ 

• ■ .- i •*, ’ ‘ ; \ 


Saturday 

visitors 

survey. 

Cohort 

feedback 


Random 

survey 

feedback 


Saturday 

visitors' 

survey. 

Cohort 

feedback 


Random 

survey 

feedback 


Extended School 
Members 
(Sphere 2) 


'V ' V ? • - ”■ '• - i > S' % . 


Cohort survey, 
random 
feedback, 
artifacts 


Cohort survey, 
random 
feedback, 
artifacts 


Cohort survey, 
random 
feedback, 
artifacts 


Cohort survey, 
random 
feedback, 
artifacts 


Extended 
Community 
Members 
(Sphere 3) 


..‘Vs ; ;■ . i ^ 5 v. ' - 




Visitors' 

survey 


Visitors’ 
survey, non- 
visitors’ 
survey, 
community 
indicators 


Visitors' 
survey, non- 
visitors' 
survey, 
community 
indicators 
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Evaluation Standards 

The American Evaluation Association (AEA), a professional organization 
dedicated to implementation and advancement of state of the art evaluative 
techniques in various settings, has created Program Evaluation Standards. These 
benchmarks for evaluators are used to ensure that evaluation provides accurate, 
valid and reliable information that meets the needs of the intended stakeholders. 
It will be the duty of the evaluation team to ensure that these Standards are fully 
implemented and utilized as appropriate for the purposes of this project. Major 
themes of the Program Evaluation Standards include the utility (validity), 
feasibility, propriety, and accuracy (reliability) of the evaluative activities. 

The evaluation plan for this project supports these Standards to ensure that the 
information provided for the various stakeholders is valid and reliable. It is the 
duty of the Evaluation Team to ensure that these Standards are fully implemented 
and utilized as appropriate for the purposes of this study. A summary of the 
Program Evaluation Standards can be found in Appendix A of this document. 




Gathering some 
evidence at the All 
About Me computer 
kiosks. 
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Results 



Formative Data Year 1 

During the first year of the study, instrumentation was developed to collect data 
regarding program implementation issues and concerns. The data was formative 
in nature, and was intended to provide guidance for the HealthWorks! staff in 
the daily operation of the museum, along with some sense of progress being 
made toward attainment of the four program goals. 

The data measures collected in Year 1 of the study included: 



•f student feedback, 

S student artifacts, 

V student customer satisfaction survey data, 

•S teacher feedback, 

S teacher customer satisfaction survey data, 

V cohort group feedback, 

S staff feedback, 

•S staff artifacts, 

■S attribute data (e.g. attendance figures), and 

V feedback from external evaluators. 



Customer Satisfaction Survey 

It was decided to collect formative information about programming at 
HealthWorks! through a customer satisfaction survey. This survey, designed by 
various Memorial staff, captures data and feedback related to the following three 
groups of stakeholders: 1) teachers attending HealthWorks! with their students; 
2) students attending HealthWorks! with their school group; and 3) drop-in adult 
and student visitors. 

In Year 1 of the study, it was decided to survey only a portion of the more than 
700 schools visiting the museum. A random stratification process was utilized to 
survey a heterogeneous sampling of the attendees. Actual copies of the surveys 
can be found in Appendices B & C at the end of this report. The overall design of 
the survey was a Likert scale using a 1 through 5 rating of various statements 
related to programming, exhibits usage and staff expertise. An open-ended 
response format was also utilized to generate feedback related to these and other 
museum issues. 

Teacher response to the items can be found on the page 17 of this report. The 
five-point Likert score has been transformed to a scaled score, with the mean 
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Mean Score 


Sample = N 


Overall Teacher Survey Rating 


94.68 


N = 36 








Overall How Things Worked Rating 


92.50 


N = 36 


Ease of Finding Way Around 


92.36 


N = 36 


Instructions on the Exhibits 


88.19 


N = 36 


Cleanliness of the Museum 


97.92 


N = 36 


Wait Time to Use Exhibits 


92.36 


N = 36 


Working Order of Exhibits 


91.43 


N = 35 








Overall HealthWorks! Staff Rating 


97.57 


N = 36 


Visibility of Employees 


95.14 


N = 36 


Friendliness of Employees 


100.00 


N = 36 


How Questions Were Answered 


97.79 


N = 34 


How Student Questions Were Answered 


98.53 


N = 34 


» t?&&h «»5ES»KA-i* t:^r it 


C C* *J f iyA 




Overall Class Rating 


97.38 


N = 35 


Interest Level of Class 


95.71 


N = 35 


Quality of Class Teacher 


99.29 


N = 35 


Subject Matter 


97.14 


N = 35 






- ".<■?■ . ■‘r. 4 - 


Overall Things to Do Rating 


91.55 


N = 36 


How Well Pre-trip Information Prepared You 


82.50 


N = 30 


Interest Level of the Exhibits 


94.12 


N = 34 


Likelihood of Recommending HW! To Others 


97.92 


N = 36 


Knowledge Student Gained 


92.14 


N = 35 




o *z '. . ” -v j, * ' .. ■ 

*?, 


..V. 
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score reported for the sampling of 36 teachers. Responses by teachers were 
extremely favorable to the museum’s programming, exhibit floor and staff. 
Highest scores were generated for staff efforts and classroom presentations. 
Some concern was expressed regarding pre-trip information provided for 
teachers. 





Mean Score 


Sample = N 


Overall Kids Survey Rating 


90.39 


N = 524 






i &- r. ~ * 


Overall How Things Worked Rating 


86.51 


N = 521 


Ease of Finding Way Around 


90.20 


N = 513 


Instructions on the Exhibits 


88.46 


N = 509 


Cleanliness of the Museum 


91.93 


N = 511 


Wait Time to Use Exhibits 


67.12 


N = 501 


Working Order of Exhibits 


95.78 


N = 504 






0“r *. :.i h&ijSfrf*** *r 


Overall HealthWorks! Staff Rating 


92.65 


N = 518 


Visibility of Employees 


88.70 


N = 511 


Friendliness of Employees 


96.53 


N = 511 


How Questions Were Answered 


93.37 


N = 483 








Overall Class Rating 


90.80 


N = 515 


Interest Level of Class 


93.27 


N = 505 


Quality of Class Teacher 


94.49 


N = 499 


Subject Matter 


84.41 


N = 489 


^xx^T:U <k 




* V .7" 


Overall Things to do Rating 


92.25 


N = 512 


Interest Level of the Exhibits 


93.43 


N = 506 


Quantity of What I Learned 


90.81 


N = 506 


Likelihood of Recommending HW! To Others 


92.73 


N = 492 
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The chart on page 18 describes the data produced from the student portion of the 
customer satisfaction survey. Data related to a randomly stratified sampling of 
524 students of the 14, 518 students attending with their school groups in the first 
year of the museum’s opening is provided. Students expressed overall positive 
comments for their museum experience, with the highest scores related to both 
HealthWorks! staff and the exhibits themselves. Students had their least positive 
feedback concerning wait time for use on the exhibit floor. 

A comparison of teacher and student responses for the same questions appears 
in the chart on page 20. It is interesting to note that though teacher and student 
view of wait times for exhibits differ, extremely positive feelings for overall staff 
and.program effectiveness do coincide. This is sconfirming evidence that both 
teachers and students attending HealthWorks! strongly support the 
appropriateness of the programming offered and the overall staff effort to deliver 
a fun and exciting way to discuss healthy living. 

Open-ended Responses 

A sampling of the open-ended response items form the customer satisfaction 
survey provided the following type of feedback: 



Fun 

□ THE EXHIBITS WERE REALLV FUN. 

□ The museum was great. I had a blast. 

□ I HAD LOTS OF FUN l 

Evhibits 

□ The exhibits were fun. 

□ the exhibits were wonderful. 

□ the exhibits were cool. 

Staff 

□ the people were nice and friendlv. 

□ I think the Egyptian guv was strange. 

□ the people are verv helpful. 




Overall Feedback 

The overall feedback from the open-item responses was also extremely positive 
towards the staff and museum programming. The chart on page 20 compares 
the overall ratings by teachers and students attending the museum. 
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Mean Score 
Teachers 


Mean Score 
Students 


Overall Survey Rating 


94.68 


90.39 



Overall How Things Worked Rating 


92.50 


86.51 


Ease of Finding Way Around 


92.36 


90.20 


Instructions on the Exhibits 


88.19 


88.46 


Cleanliness of the Museum 


97.92 


91.93 


Wait Time to Use Exhibits 


92.36 


67.12 


Working Order of Exhibits 


91.43 


95.78 



Overall HealthWorks! Staff Rating 


97.57 


92.65 


Visibility of Employees 


95.14 


88.70 


Friendliness of Employees 


100.00 


96.53 


How.Questions Were Answered 


97.79 


93.37 


How Student Questions Were Answered 


98.53 


liljB|jl 



Overall Class Rating 


97.38 


90.80 


Interest Level of Class 


95.71 


93.27 


Quality of Class Teacher 


99.29 


94.49 


Subject Matter 


97.14 


84.41 



Overall Things to Do Rating 


91.55 


92.25 


How Well Pre-trip Information Prepared You 


82.50 


93.43 


Interest Level of the Exhibits 


94.12 


90.81 


Likelihood of Recommending HW! To Others 


97.92 


92.73 


Knowledge Student Gained 


92.14 
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Some of the responses were treated as outliers, since they expressed a viewpoint 
not confirmed by others. Responses that were able to be chunked into categories 
portraying a certain theme or concern were duly noted. At this time the sample 
size of the drop-in visitors survey was deemed insufficient to report. It is 
hopeful that the data set will be large enough to comment on in the biennial 
report of August, 2001 . 



Cohort group Feedback 

The cohort group design provides an opportunity to solicit comments and 
feedback from teachers who have attended the museum on more than one 
occasion. This group is recognized as a highly credible source for formative 
assessment feedback used to improve current program offerings and the overall 
performance of the museum. 

During Year 1 of the study, teachers were asked to comment on the relative 
strengths and challenges of the museum activities, along with comments for 
future programming. The cohort sample size was 13, and again produced 
extremely favorable comments for both the museum experience and the staff. 
The comments were compiled through fax and e-mail responses to structured 
questions found in Appendix E of this report. 

A small sampling of some of the feedback from the cohort teachers is found in 
the chart below: 



classroom Presentations 

• VERV ORGANIZED AND INFORMATIVE. 

• THE KIDS ARE UP AND MOVING AROUND. 

• THE CLASSROOM PROGRAM PRESENTATIONS ARE VERV ENGAGING. 

Eyhibits 

• Excellent! Great wav of having students explore. 

• THE EXHIBITS ARE GREAT. 

• LOVED THEM! 

Staff 

• EvERVONE IS VERV FRIENDLV. 

• 1 FEEL VOU DO A GREAT JOB. 

• THE TEACHERS ARE VERV CLEAR AND INFORMATIVE. 
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Staff feedback 

Informal commentary from staff on programming and general management of 
the museum was routinely gathered through monthly staff meetings. Other 
more formal feedback was illidted after the museum was opened six months and 
at the end of the first year. The staff comprises a focus group in and of 
themselves, but their comments and feedback will be tracked and compared with 
other focus groups noted in the study. Initial feedback from the staff in Year 1 is 
very positive and staff generally expresses satisfaction as a group. More formal 
comparison of staff feedback with other focus group is to be completed as part of 
the biennial report. 



Artifact review 

Many students have provided unsolicited 
feedback through cards, letters and drawings 
expressing their positive feelings and attitudes 
about their experience at HealthWorks! These 
artifacts provide evidence that programming is 
both fun and interesting to the students. A 
sampling of student artifacts is always on display 
at the museum on the wall located behind the 
front desk. The picture on this page shows a 
sampling of some of the student artifacts received 
by the staff on display. 

Evaluation review 

The Evaluation Team has formally solicited 
feedback on planning, implementing and revising 
the formal evaluation plan for the five years of the 
study. A meeting was held in Indianapolis, IN in 
the fall of 2000 to gather commentary on the initial 
program evaluation proposal. Noted program 
evaluators from across the nation have shared 
their expertise and ideas about the five-year study. 

Subsequent additions and revisions incorporated in to the study reflect the best 
practices and thinking of the larger evaluation community. Evaluators continue 
to be apprised on issues concerning data collection, methodology and data 
analyses. This dialogue ensures that the evaluation plan is valid and reliable, 
and also provides for opportunities to envision new ways to envision and 
conduct program evaluations. In subsequent years of the study, additional input 
will be formally solicited as needed through external evaluators versed in both 
health program evaluations and museum studies. 




Just some of the 
thank you notes 
and letters sent 
to us by 
students and 
proudly 
displayed at 
HealthWorks! 
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Summary of Findings 

Formative data collected in Year 1 of the study was designed to gather feedback 
on program implementation and initial feedback for future planning. The data 
sets included in Year 1 of the study and described previously in this report are: 

> Customer satisfaction survey from teachers, 

> Customer satisfaction survey from students, 

> Staff feedback, 

> Cohort teachers group feedback, 

> Artifact review, and 

> Evaluation review. 

Triangulation of these data sets was combined with the confirming evidence 
approach to gauge the extent to which current program goals have been met. 
The interpretation of the data is represented in the following chart: 

Evidence Chart 

Program Goals : Student Focus 



Goal 


Children participating in 
HealthWorks! will: 


Confirming 

Evidence 


Mixed 

Evidence 


Disconfirming 

Evidence 


1 


engage in interactive processes of 
discovery that will stimulate the 
curiosity, and result in new 
exploration, knowledge, and 
positive attitudes about healthy 
living. 


V 






2 


be intrigued and empowered to 
explore the many choices and 
options they have and will develop 
the resources and skills to make 
informed choices in their daily 
living. 


V 






3 


appreciate the wonder and 
complexity of the human body, its 
senses, intelligence and spirit, and 


V 






4 


remember the visit as fun, exciting 
and meaningful, will extend their 
exploration of healthy living beyond 
their experience, and will return 
with family and friends. 


V 
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Claim Statements 

Based upon the evidence presented through the formative assessments the 
following claim statements are said to be true and relevant. These claim 
statements are categorized by the corresponding spheres of influence. Some 
sample claim statements from data collected in Year 1 of the study follow: 



Sphere 1 Children 

Confirming evidence exists that children: 

• engage in interactive processes of discovery at the museum, 

• explore the many choices they have in their daily living, 

• appreciate the wonder and complexity of the human body, 

• remember the visit as fun, exciting and meaningful, and 

• will return with family and friends. 



Sphere 2 Family & School 

Confirming evidence exists to support: 

• teacher, parents & community members have increased awareness and 
knowledge of health issues through participation at the museum. 



Sphere 3 Community 

Confirming evidence exists to support: 

• the dream of a planned program of interactive and meaningful health 
education curricula & instruction has been realized. 




Hittting the 
mark with the 
HealthWorks! 
exhibits. 
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Conclusions & Recommendations 

The program staff of HealthWorks! Kids' Museum, Memorial Health Systems 
and the Memorial Health Foundation are to be commended for envisioning and 
bringing to fruition a truly interactive museum in which students can explore 
and discover the choices they face regarding healthy living. 

Recommendation #1 

Continue data collection and analyses procedures for formative 
assessment measures for use in program planning and implementation, 
such as: 

♦ Customer satisfaction survey from teachers, 

♦ Customer satisfaction survey from students, 

♦ Staff feedback, 

♦ Cohort teachers group feedback, 

♦ Artifact review, and 

♦ Evaluation review. 

Recommendation #2 

Refine program goals to include program objectives related to existing 
and new program offerings. 

Recommendation #3 

Begin analyses of summative data collection assessment procedures 
related to the following data sets: 

♦ student feedback, 

♦ student artifacts, 

♦ student customer satisfaction survey data, 

♦ teacher feedback, 

♦ teacher customer satisfaction survey data, 

♦ cohort group feedback, 

♦ staff feedback, 

♦ staff artifacts, 

♦ attribute data related to HealthWorks! and 

♦ feedback from external evaluators. 

Recommendation #4 

Continue collection and tabulation of all attricbute and frequency data 
archived in the electronic scheduling, classroom voting, and All About Me 
computer kiosks, for sy tematic study and long-term summative analyses. 

Recommendation #5 

Begin design and collection of community health program indicators for 
long-term summative analyses. 
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Appendix A 



THE PROGRAM EVALUATION STANDARDS 
Summary of the Standards 

Utility Standards 

The utility standards are intended to ensure that an evaluation will serve the information needs of intended 
users. 

U1 Stakeholder Identification Persons involved in or affected by the evaluation should be identified, so that 
their needs can be addressed. 

U2 Evaluator Credibility The persons conducting the evaluation should be both trustworthy and competent 
to perform the evaluation, so that the evaluation findings achieve maximum credibility and acceptance. 

U3 Information Scope and Selection Information collected should be broadly selected to address pertinent 
questions about the program and be responsive to the needs and interests of clients and other specified 
stakeholders 

U4 Values Identification The perspectives, procedures, and rationale used to interpret the findings should 
be carefully described, so that the bases for value judgments are clear. 

U5 Report Clarity Evaluation reports should clearly describe the program being evaluated, including its 
context and the purposes, procedures, and findings of the evaluation, so that essential information is 
provided and easily understood. 

U6 Report Timeliness and Dissemination Significant interim findings and evaluation reports should be 
disseminated to intended users, so that they can be used in a timely fashion. 

U 7 Evaluation Impact Evaluations should be planned, conducted, and reported in ways that encourage 
follow-through by stakeholders, so that the likelihood that the evaluation will be used is increased. 



Feasibility Standards 

The feasibility standards are intended to ensure that an evaluation will be realistic, prudent, diplomatic, and 
frugal. 

FI Practical Procedures The evaluation procedures should be practical, to keep disruption to a minimum 
while needed information is obtained. 

F2 Political Viability The evaluation should be planned and conducted with anticipation of the different 
positions of various interest groups, so that their cooperation may be obtained, and so that possible attempts 
by any of these groups to curtail evaluation operations or to bias or misapply the results can be averted or 
counteracted. 

F3 Cost Effectiveness The evaluation should be efficient and produce information of sufficient value, so that 
the resources expended can be justified. 



Propriety Standards 

The propriety standards are intended to ensure that an evaluation will be conducted legally, ethically, and 
with due regard for the welfare of those involved in the evaluation, as well as those affected by its results. 

PI Service Orientation Evaluations should be designed to assist organizations to address and effectively 
serve the needs of the full range of targeted participants. 

P2 Formal Agreements Obligations of the formal parties to an evaluation (what is to be done, how, by 
whom, when) should be agreed to in writing, so that these parties are obligated to adhere to all conditions of 
the agreement or formally to renegotiate it. 

P3 Rights of Human Subjects Evaluations should be designed and conducted to respect and protect the 
rights and welfare of human subjects. 
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Appendix A (contd. ) 



P4 Human Interactions Evaluators should respect human dignity and worth in their interactions with other 
persons associated with an evaluation, so that participants are not threatened or harmed. 

P5 Complete and Fair Assessment The evaluation should be complete and fair in its examination and 
recording of strengths and weaknesses of the program being evaluated, so that strengths can be built upon 
and problem areas addressed. 

P6 Disclosure of Findings The formal parties to an evaluation should ensure that the full set of evaluation 
findings along with pertinent limitations are made accessible to the persons affected by the evaluation and 
any others with expressed legal rights to receive the results. 

P7 Conflict of Interest Conflict of interest should be dealt with openly and honestly, so that it does not 
compromise the evaluation processes and results. 

P8 Fiscal Responsibility The evaluator's allocation and expenditure of resources should reflect sound 
accountability procedures and otherwise be prudent and ethically responsible, so that expenditures are 
accounted for and appropriate. 



Accuracy Standards , . „ 

The accuracy standards are intended to ensure that an evaluation will reveal and convey technically 
adequate information about the features that determine worth or merit of the program being evaluated. 

Al Program Documentation The program being evaluated should be described and documented clearly and 
accurately, so that the program is clearly identified. 

A2 Context Analysis The context in which the program exists should be examined in enough detail, so that 
its likely influences on the program can be identified. 

A3 Described Purposes and Procedures The purposes and procedures of the evaluation should be 
monitored and described in enough detail, so that they can be identified and assessed. 

A4 Defensible Information Sources The sources of information used in a program evaluation should be 
described in enough detail, so that the adequacy of the information can be assessed. 

A5 Valid Information The information-gathering procedures should be chosen or developed and then 
implemented so that they will assure that the interpretation arrived at is valid for the intended use. 

A6 Reliable Information The information-gathering procedures should be chosen or developed and then 
implemented so that they will assure that the information obtained is sufficiently reliable for the intended 

use. 

A 7 Systematic Information The information collected, processed, and reported in an evaluation should be 
systematically reviewed, and any errors found should be corrected. 

A8 Analysis of Quantitative Information Quantitative information in an evaluation should be appropriately 
and systematically analyzed so that evaluation questions are effectively answered. 

A9 Analysis of Qualitative Information Qualitative information in an evaluation should be appropriately 
and systematically analyzed so that evaluation questions are effectively answered. 

A10 Justified Conclusions The conclusions reached in an evaluation should be explicitly justified, so that 
stakeholders can assess them. 

All Impartial Reporting Reporting procedures should guard against distortion caused by personal feelings 
and biases of any party to the evaluation, so that evaluation reports fairly reflect the evaluation findings. 

A12 Metaevaluation The evaluation itself should be formatively and summatively evaluated against these 
and other pertinent standards, so that its conduct is appropriately guided and, on completion, stakeholders 
can closely examine its strengths and weaknesses. 



0 
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Appendix B 



Student Survey Questions 



How are we doing Kids? Survey 



INSTRUCTIONS: You rcccndy wished Hokfi Worts! Kids Museum. Phase fi 
^ wrl c - r Chtk the ompbor that ben ivptuu o your feefaty- If you had no c 



. THANK TOU1 
sot HmWiWoHcd 




e here with: 

A Ooxs from school 



Other_ 



_yei«oid 



n the bacfcptuid questions and then rate year 
a item, skip is. AbocaonmKon 
f, please nail it tn the 



lent: OAgH □ A Boy I Qf 

This arey was filled out: W, 

□ All by myself 

□ With help from My teacher 

□ With help fraei a parent 

□ With help from an alder child 



Mow things worked gt Hy qfrhWocfai 

1. Finding ny way to exhibits and other places was. *ta*yMwd 

2. Instruction* on the exhibits were Ihd to MU 

S. HcahhWoHal Tidf Msrsctnt and the exhibits were IrkydWy 

4 Getting My t»m te as* the exhibits X wanted to took A long Hew 

Sl The exhibits I wanted to ase were fcehwi.eet *f • 



12 3 4 
12 3 4 
12 3 4 
12 3 4 
12 3 4 



Easy t* teflon 



Nottwe at 

Mhriwdfe* 



Y 

The Peopk Who Woric at HtqlthWqdg l 

L The petpie et HeaHhWarfcd were' 

2. The pespla wha work at HeehhWcrtcd are Not 

X When I asked the HtahhWarkd guestiom. 6at» 



satafl 
low’d d 



W 

1 2 3 4 5 J 
1 2 3 4 5 Very Mce A Friendly 
1 2 3 4 3 A chnrgednow 



m 



Qytrdl. tht Thing* ft Dq at HwIthWortaJ ^ 

L The exhttts here are Totally baring 1 2 3 4 5 

2. The doss we went to was Not wery interesting 1 2 3 4 5. 

3. I tie** I lamed. hbthn,atafl 1 2 3 4 5 

4 How l*ely <m» I to n eco mw wn d Health Wcrkd to friends?. . htot at afl 1 2 3 4 5 

These ere the things I bleed best at HeaHhWaHcd 



Vvy 

TataOy exciting 
Very interesting 
Lots 



The mast ieyertant thing X learned ■ 



One Ihwg I wiO be sn to do to Ue a healthy life te 
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APPENDIX C 

Student Performance Record Sheet 



Indivitlunl Record TEACHER: 

Qet a Jump Ou It ! - How high did you jump? inches 

Celling the Hang of It - How long Can You Hang? : 

The Real Goes On - Your high RPMs Your Highest CLS/hour 

Did you make it all the way across the Skin Crawl Wall without dropping? YES NO (circle one) 

pow Fast Can You React? - Place your hand on the table. Press the START button and attach the yardstick to the magnet. Watch the light. 
Grab the yardstick when it drops. Keep your hand on the table. Circle where your hand grabbed the stick , GREEN YELLOW RED 

What Can You Tell bv Touch? “ left box, 1 guessed 1 2 3 4 5 correctly, right box, I guessed 1 2 3 4 5 correctly (circle) 

Healthy Pacts Hunt 

See how many facts you can find. One point for every correct answer. Not correct without the UNITS, like feet, inches, tons etc. 

How tall was the shortest man in the world? ; How many different smells can you nose identify? — 1 

About how long does it take your body to digest your lunch? '' ' What is your biggest organ? . 

How long would it take to smoke all the cigarettes in the chair? 

How fast can the nerve impulses travel? How many different smells can you nose identify? 

What is the medical word that means food going down your tliroat to your stomach? 

Name the only smell item, at the Remember This Smell station, you should not put in your mouth 1 : 

What would you have if your ‘snot’ were yellow? - ; 
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Annual Report of Findings 



Executive Summary 
January, 2001 




Dennis W. Rudy, PhD. 
Indiana University South Bend 



urn, which opened on 
12,000 square foot 



Overview 

HealthWorks! Kids' Musei 
February 12, 2000, is 
innovative education center located within the 
Memorial : Leighton HealthPlex in downtown 
South Bend, IN. It is designed to help children 
in grades PreK-8 understand and make good 
choices about healthy living and lifestyle 
choices. The museum is open daily for school 



groups, and evenings and 
community groups and families: 



weekends for 



The museum was proposed, designed and 
implemented in response to the needs of the 
community. Countless, community members. 



on health-related questions, 

♦ The Skin/Crawl Wall - a professional 
climbing wall that looks like magnified 
human skin, 

•' Virus Invaders - a live action video game 
where one fights off bacteria and viruses,. 

♦ Interactive Video Classrooms - for 
demonstrations and group activities, and 

♦ Resource Center - offering reference . 
materials for students, teachers and families. 



consultants and specialists from across the 
nation in determining program offerings, exhibit 



designs, 

for the museum. • 

The museum consists of an exhibit floor and 
interactive classroom areas. Some features, of 
the HealthWorks! Kids’ Museum currently 
include: . 

♦ Body Works! - highlighting body systems and 
how they work, 

4 The Main Brain, theatre - an exploration into 
. the mind, ■ * ; 

♦ MihdWorks! r challenging the senses and 




O 

ERiC 



8 • * 


► All About. 


Me VI 


aosksof con 


iputer 


s focus 


sing 

i 

















The HealthWorks! Kids' Museum is just one of 
many community health outreach efforts 
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Memorial Health Foundation, such as bicycle 
safety, anti-smoking, and sexual abstinence 
programs for students in elementary, middle 
and high schools. The doctors and staff of 
Memorial Hospital of South Bend have; a long 
and rich history of community, involvement 
within the Michiana region’ . . 

The project is sponsored by Memorial Health 
Systems and the Memorial Health Foundation, 
along with funding by the Harry and Jeannette 
Weinberg Foundation, Dennis and Mary Lou 
Schwartz, James and Julia Schwartz, the Shields 
and Warner families, the Memorial Hospital 
Auxiliary, and the Junior League of South Bend; 

Spheres of Influence 

The. founders of Health Works! envision an ever 
widening sphere of influence beginning with the . 
museum activities. It is hoped that once a child 
attends the museum, she or he will want to 
share the museum experience with others 
important in their lives. Thus, it is believed that 
family members and others at school (i.e. 
teachers, staff & students) will be made aware of 
healthy living ideas and opportunities for them 
in which to share. .' 

Eventually this ever widening sphere of 
influence would impact the community in 
positive ways related to healthy living. The 
diagram below describes this three-tier model of 
influence, with Sphere ! related to children who 
attend HealthWorks! Kids' Museum, Sphere 2 
consisting of family and school members, and 
Sphere 3 representing the larger community of 
Michiana. : • • : • • • ■ ’ ; • : • ’ ' ; • 




Confirming Evidence 

The. evaluation plan utilized a confirming 
evidence approach to gauge program 
effectiveness, and provide insight and 
understanding for continued planning of the 
partnership activities for the duration of the 
. project. For example, each program goal and 
research question addressed was reviewed and 
studied both formative] y and summati vely, thus 
providing feedback on current and past project 
efforts along with . recommendations for 
addressing long-term program planning issues 
(Patton, 1.990; W or then & Sanders, 1994). 

Each program goal was said to have been 
achieved or met by Confirming Evidence, if and 
when multiple data measures confirmed same. 
Program goals and research questions which 
produced conflicting or differing pieces of data 
measures were said to have produce Mixed 
Evidence , and were deemed inconclusive. 
Disconfinning Evidence was the term, used when , 
multiple data measures verified that program 
goals have not been successfully met (Rudy,. 
-1999a; Rudy, 1999b). 




Multiple Assessments 



A method of data collection used by many 
researchers to ensure accurate findings is called 
triangulation; This procedure requires that 
multiple data measures (i.e. minimally three 
distinct measures) be used to produce evidence 
; related to each of the questions addressed in the 
evaluation plan (Miles & Huberman, 1984; 
Denziit & Lincoln, 1994). This variety and 
multiplicity of data measures strengthens the 
validity and reliability of any conjectures 
ultimately made regarding progress made, 
toward, achieving the program goals. The 
. evaluation, plan, for this project, integrated 
triangulation with the . confirming evidence 
approach as the base of the evaluation activities. 
Latest findings from the research community 
support a mix. of qualitative and quantitative 
data. . sources when ^conducting ! program 
evaluation research (Frechtling & Sharp, 1997). 































Data Collection Plan 

The evaluation plan encompasses both content 
and process concerns (i.e. health content 
: knowledge and behavior; attitude, instruction), 
formative and summative assessment (i.e. short- 
term and long-term planning/ feed back), and 
on-site and off-site activities (i.e. health center . 
and non-museum activities). A myriad of data 
measures are to be utilized, some naturally 
occurring in the project environment and some 
instrumentation that will have. to be created and 
implemented: Remember that triangulation is to . 
. be utilized to verify the extent to which program 
goals are met, so data will be abundant and 
complex, 

Several broad categories of data measures were 
utilized in data collection and data analyses. 
Again both qualitative and quantitative 
measures were utilized to assess the extent to 
which program efforts successfully addressed 
the three foci, of students,, staff and parents 
described in the major project goal. 

The following performance indicators were 
selected and deemed appropriate for use in the 
study by the project staff and oversight 
committees. The indicators are categorized by 
the three spheres of influence: 

Sphere 1 Children 

• Observations of Students at Health Works! 

• Student Surveys 

. •• Student Artifacts 
; •' Student Feedback 

• Student Performance at Health Works! and in 
related classroom activities. 

Sphere 2 Family & Teachers 

; •- Survey of Saturday Visitors 

• Feedback from Parents / Families Whose 
Children Attended Health Works! 

• Teacher Surveys 

• Teacher Artifacts 

• Teacher Feedback 




Evaluation Timeline 

Phase 1 Formative Evaluation 
Year 1 of the Study 
January, 2000 - December 31, 2000 

Phase 2 Summative Evaluation 
Years 1 & 2 of the Study 
January, 2000 - August, 2001 . 

Phase 3 Formative Evaluation 
Year 3 of the Study 
August, 2001 - August, 2002 



Sphered Community 

• Community Surveys 
Community Feedback 

. • Community Artifacts 

• Community Health Related Data (trends, 
patterns, pre-posttests) 



Phase 4 Summative Evaluation 
Years 3 & 4 of the Study 
August, 2001 - August, 2003 

Phase 5 Summative Evaluation 
Years 1 - 5 of the Study 
January, 2000 - August, 2004 





















Findings & Implications 

Some sample claim statements from data 
collected in Year .1 of the study follow:. ; 

Sphere 1 Children 



Confirming evidence exists that .children: 

engage in interactive processes of discovery 
at the museum, 



explore the many choices they have in 
daily living, ; 

appreciate the wonder and complexity 
the human body, . 

remember the visit as fun, exciting 
meaningful, and 

will return with family and friends. 
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Sphere 2 Family & School 

Confirming evidence exists to support: 

♦ teacher, parents & community members 
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Program Offerings 

Current program offerings in the second year of 
the health museum operation are: 

Safety : You've Got What It Takes! . ’ . 

Students , review common personal safety rules 
through role-playing, demonstration and games. 

Mission Possible: A Healthy Body! - : 

Students investigate nutrition, exercise, rest, and 
attitude as they strive to uncover the mystery of 
vyell-being. 

The Amazing Body 
Students examine the heart, lungs, brain, bones 
and muscles to determine which is the body 
superstar through working on teams. 
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have increased awareness and knowledge of 
health issues through participation at the 
museum 



Sphered Community 



Confirming.evidence exists to support: 

♦ ‘ the dream of a planned program 
interactive and meaningful health educa 
curricula & instruction has been realized 
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Want More 
Information? 



Deborah Drendal!/ Director, 

. Hea lth W orks! Ki d s’ Muse urn; 
219-287-KlDS • • 



ddrenci all ©quality of iife.org 



Dennis W. Rudy, External 
Evaluator,- 
616-469-7946 . 



drudy@lakehouse.org 
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